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ABSTRACT 

What should the curriculum of a graduate program that focuses 
on education, technology, and society be? What would be the proper balance of 
skills, theory, and concepts? What should be the difference between such a 
program in the School of Education and a computer science program? What do 
students expect of such a program? These were questions with which the 
faculty of the Technology for Education and Training Program at the 
University of South Dakota wrestled with in spring 2001, at a time when the 
Master's and Specialist's degrees had been offered for 2 years. An 
"appreciative" program evaluation was conducted to address the questions the 
faculty had about the curriculum and to ascertain how the program was doing 
in general. This paper describes the process used to assess the Technology 
for Education and Training graduate program. The appreciative evaluation was 
designed to discover what was working well in the program and how offerings 
could be improved. The program evaluation was completed by 445 current and 
former students. Eight student evaluators interviewed faculty members and 
analyzed survey and interview data. The evaluation approach showed many 
things that were being done well and also resulted in recommendations for the 
improvement of the program. (Contains 36 references.) (SLD) 
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Abstract 



What should the curriculum of a graduate program that focuses on education, technology 
and society consist of? What would be the proper balance of skills, theory, and concepts? 
What should be the difference between such a program in a School of Education vs. a 
computer science program? What do students expect of such a program? These were 
questions the faculty of the Technology for Education and Training Program at the 
University of South Dakota were wrestling with Spring 2001 . At this point, the Master's 
and Specialist degrees had officially been offered for two years. An "appreciative" 
program evaluation was conducted to address the questions the faculty had about the 
curriculum and to ascertain how the program was doing in general. This paper describes 
the process used to assess the Technology for Education and Training graduate program. 



Introduction 

In Spring 2001, the Technology for Education and Training Division at the University of 
South Dakota assessed the status of the current degree offerings and programs. At this 
point, the Technology for Training and Development (TTD) Master’s and Specialist 
degrees had been officially offered for two years. The Master’s degree had two tracks 
within it: one with a K-12 emphasis and one with a Training and Development emphasis. 
The Program was created to appeal to both K-12 teachers and those from non-school 
settings. K-12 students might be attracted to the Program to sharpen their skills in 
technology to prepare for becoming technology coordinators in their schools or simply to 
learn better how to integrate technology into their curriculum. Students in the Training 
and Development track might be attracted to gain skills to become instructional 
designers, trainers, or organizational development specialists in business and industry. 

At this two-year mark, the faculty determined an assessment of the status of the program 
offerings was in order. They sensed some confusion on the part of students as to what the 
program was really about. Was it technology skill development? Was it more 
conceptual than that? What did it mean to develop leaders in the appropriate use of 
technology in school and non-school settings? 

Eight students enrolled in Spring semester courses were recruited to help with this 
project. The assessment was an appreciative inquiry: an inquiry to discover what was 
working well in the program offerings and how the offerings could be improved 
(Cooperrider & Whitney, 1999a). Rather than an inquiry into what wasn't working to 
determine what there should be less of, it was an inquiry into what was working and what 
there should be more of: getting rid of what we do not want does not mean we will get 
what we do want (Ackoff, 1999). It was also an action research project: the emphasis 
was on developing practical knowledge while engaging stakeholders (students) in the 
questioning, collection of data to address those questions, and sensemaking of the data 
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collected (Reason & Bradbury, 2001). The project employed qualitative research 
methods and was conducted under the guidance of two Division faculty members. 



Framing Evaluation Appreciatively 

Appreciative Inquiry (AI) is a "cooperative search for the best in people, their 
organizations, and the world around them” (Cooperrider & Whitney, 1999a, p. 10). AI 
differs from traditional organizational interventions in that it is not a method of problem 
solving. Instead of searching for the problem to solve, it begins with a search for the best 
of what is. It is a search to discover the life-giving forces of the system and to identify 
what the system wants "more" of. The Program faculty wanted to know what the 
students valued about the current program structure and offerings and what we should be 
doing “more” of. By using an evaluative approach that focused on what was working, we 
would gain information about the strengths of the program; this in turn would create 
positive forward momentum. We also decided to make this a project for a Program 
Evaluation course co-taught by two Division faculty members that semester. By making 
it a course project, students would gain practical experience in conducting a program 
evaluation and would also be engaged as stakeholders in the Program. The goal was to 
discover the best of what we had and use it to create a "collective image of a desired 
future," (Mohr, Smith & Watkins, 2001, p. 292) engaging both faculty and student 
stakeholders. 

Appreciative Inquiry provided the framework for the program assessment. This is a 
framework based in the idea that every system has good and bad in it; we typically focus 
on the bad. AI provides a structure for searching out the "goodness" in the system. Four 
basic types of questions are crafted when using the AI framework. The questions are 
crafted to elicit the "best of' the current system and to understand how these are "life- 
giving" factors. They are stated in the affirmative, using positive language; are presented 
as an invitation; evoke storytelling; are phrased in the vernacular; are sometimes 
ambiguous; direct us to value what is (Cooperrider & Whitney, 1999b). The data 
collected comes primarily in the form of stories: we knew students (former and current) 
would have stories to share about their experiences in the Program and would be more 
likely to share them with a fellow student rather than a faculty member. Both the 
Division faculty and the students who conducted the actual assessment crafted the 
questions used for the program evaluation. 

To set the storytelling mode, the first type of question asked in AI is a "deep story 
question" (Cooperrider & Whitney, 1999b). This question asks the interviewee to tell a 
story about a peak experience or high point. They are encouraged to describe who was 
involved, what made it a peak experience, what they did to make it a peak experience, 
what others contributed to make it a peak experience. We had two "deep story" 
questions: 

1) Why did you choose this program? 
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2) Share a highlight of your time so far in the TTD Program. As you share your story, 
consider the following: What made it a high point? Who was involved? What did they 
do that made it a good experience? What did you do that made it a good experience? 

The second type of question has to do with valuing the system (in this case, the Program) 
(Cooperrider & Whitney, 1999b). The interviewee is asked what they value about the 
system. The third type of question is the "core factors" or life-giving question: What 
gives "life" to the system? (Cooperrider & Whitney, 1999b). This question elicits the 
specifics about what gives life to the system and seeks to understand why it gives life. 
Rather than identify the causes of a problem, this question asks the interviewee to ponder 
the best of the system and understand these factors as deeply as we typically understand 
what is wrong and why it is wrong. We crafted a question that captured both of these 
concepts: Based on your experience so far, what do you value most about the Program? 
What would you define as core characteristics of the Program (without these, the 
Division and Program would not be what it is)? Please be specific. 

The fourth type of question is the "future" question (Cooperrider & Whitney, 1999b). 
This is the question that invites people to dream about the ideal future of the system. To 
elicit these ideas, we asked, If you were the “Student-in-Charge ” of this Program and 
could have three wishes for the program granted, what would you wish? How would the 
Program be different if your three wishes were incorporated into the curriculum? 

We also had some practical concerns. For instance, some of the courses in this Program 
are delivered using distance technologies. We knew there was dissatisfaction among 
some of the students with the distance delivery of courses. We also knew that distance 
delivery of courses was not going away, so we wanted to find out how this could be 
improved. 

The questions all students were asked follow: 

1) Why did you choose this Program? 

2) Share a highlight of your time so far in the Program. As you share your story, 
consider the following: What made it a high point? Who was involved? What 
did they do that made it a good experience? What did you do that made it a good 
experience? 

3) Based on your experience so far, what do you value most about the Program? 
What would you define as core characteristics of the Program (without these, the 
Program and Division would not be what it is)? Please be specific. 

4) If you were the “Student-in-Charge” of the Program and could have three 
wishes for the program granted, what would you wish? How would the Program 
be different if your three wishes were incorporated into the curriculum? 
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5) Many of the courses are delivered partially or fully using distance technologies 
such as Web CT and interactive video. What is it like to be in such a class? What 
courses have you had that are actually (or you believe would be) better delivered 
in a distance format? What role do you believe distance technologies should play 
in the future of the Program? 

On the surface, it may appear we only wanted to hear good comments and not find out 
about actual problems students may be having with the Program. That was not our intent, 
and in fact, using the AI framework, we (the faculty and students) discovered "the 
richness of data gained from the four generic questions allows many more questions to be 
answered" (Mohr, Smith & Watkins, 2001, p. 313). Particularly, as students described 
their wishes for the future of the Program, problems and criticisms surfaced. However, 
because the focus was on what was going right, it was easier to turn those problems and 
criticisms into positive recommendations. A creative, generative energy is engendered 
when using an AI approach (Norum, 2001). 



Methods 

Qualitative research methods were employed for this project. Specifically, narrative 
inquiry and action research methods were used. The interviews conducted were 
considered to be conversations with a purpose (Rossman & Rallis, 1998). The students 
conducting the evaluation were asking questions they and the faculty truly wanted 
answers to and they listened responsively to those answers. Data from a web-based 
questionnaire and interviews were analyzed by. the students who conducted the 
interviews. They identified themes and patterns that formed the basis of the evaluation 
report. 

Each interview conducted yielded a narrative: a story about the person’s experience so far 
with the Program, their hopes and wishes for its future, and perhaps their concerns. Thus, 
the students were engaged in narrative inquiry (Abma, 1999; Barone & Eisner, 1997; 
Clandinin & Connelly, 2000; Polkinghome, 1995). This is a specific form of qualitative 
research and while narrative as research may be relatively new to education, it has been 
used in other disciplines such as sociology, psychology, literary theory, anthropology and 
history for some time (Casey, 1995-96; Cortazzi, 1993; Josselson, 1993). It is quite 
natural for us to think narratively. Life is informed and formed by stories 
(Widdershoven,1993). Narratives occur naturally (Cortazzi, 1993) and help us make 
meaning of life's episodes (Clandinin & Connelly, 1994; Connelly & Clandinin, 1990; 
Daloz, 1986; Polkinghome, 1988, 1995; Reason & Hawkins, 1988; Simmons, 2001). It 
is quite common for people to explain their actions or relate an experience through telling 
a story. "We think and see in terms of stories because we are stories" (Feige, 1999, p. 87). 
Narrative inquiry is a heretical research method (Norum, 1998) because this method is a 
deliberate attempt to bring divergent points of view on issues to the forefront (Levin & 
Riffel, 1997). The form paints a different kind of picture, allowing for different and 
possibly new kinds of understandings (Barone & Eisner, 1997). 
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Because the practitioners were also the researchers, the students were also engaged in an 
action research project. Reason and Bradbury (2001) describe action research as a 
“participatory, democratic process concerned with developing practical knowing” (p. 1). 
The emphasis in action research methods is practical research based in (this case) 
qualitative methods. It takes everyday things in life and unpacks them (Noffke & 
Stevenson, 1995) by engaging people in a deeper understanding of their organization 
(Carr, 1997). A primary aim is to produce knowledge and action based on that knowledge 
that is directly useful to the group involved (Reason, 1998). Experiential knowledge is 
honored (Bray, Lee, Smith, Yorks, 2000; Reason, 1998). It is a collaborative process: the 
people who were involved in collecting the data are also involved in disseminating and 
applying results (Quigley, 1997). Those who would directly feel the effects of changes in 
the system are involved in shaping, what those changes might be. Stakeholders (in this 
case, students) are involved in the questioning, the collection of data to address those 
questions, and sensemaking of the data collected (Reason & Bradbury, 2001). 



The Process 

In designing the process to be used, the students who conducted the evaluation suggested 
posting the questionnaire on the web and inviting former and current students to complete 
and submit it to a generic e-mail address. A generic e-mail address was used instead of a 
specific person's e-mail address: there was a concern that if the completed questionnaire 
went to someone specific, that might discourage some people from filling it out. We 
created a Program e-mail address. After receiving the completed questionnaires, the 
students conducting the evaluation followed up with a phone or face-to-face interview. 
The web-based questionnaire used the questions described earlier; these questions also 
provided the foundation for the follow up interviews. 

An e-mail invitation to complete the web-based questionnaire and play a role in shaping 
the future of the Program was sent to 72 current and former Program students. Students 
who had graduated, those who had taken classes but were not currently enrolled, and 
current students were invited to participate in this program evaluation project. They were 
told that a current student would contact them whether they completed the questionnaire 
or not. If they did not want to be contacted, they had to let us know they were choosing 
not to participate. Less than six students asked to not be contacted. Some students could 
not be contacted — the contact information we had turned out not to be valid. In the end, 
44 students participated in the Program evaluation. 

One of the challenges with qualitative research is not so much in collecting the data, but 
what to do with it once you have it! Each of the eight students who conducted the 
Program evaluation was assigned nine students to contact or follow up with. As 
mentioned above, some students could not be contacted and in the end, data was collected 
from 44 students (including the students conducting the Program evaluation). The task 
was to take the interview data from 44 people and move from individual stories to a co- 
created story. This was done by searching for themes and refrains (Lawrence- Li ghtfoot 
& Davis, 1997): an idea that runs through all or most of the data or one idea with heavy 
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impact. Patterns were identified by looking for repeated relationships. Each of the eight 
students analyzed their interview data on an individual basis; then a group analysis of the 
data was conducted. The process was similar to that of reconstructing a jigsaw puzzle: 
puzzle pieces were identified within each individual data set and each data set became a 
piece of the overall evaluation project puzzle.. The findings that emerged from this 
analysis provided the framework for the recommendations and report that were created. 

All students participated in writing the report. A smaller group of students presented the 
results of the report to the Division faculty. The Division Faculty met within a month of 
receiving the report recommendations and identified actions to take and commitments to 
make. 



Living the Process: A Story about the Stories 

According to Liebler (1997), “The process of doing the appreciative interviews is as 
important as the data collected, for it is through the doing that the internal conversations 
within organizations are changed.” The student interviewers themselves were changed in 
carrying out the interviews as they gained new knowledge of the how the respondents felt 
about the TTD Program, and in turn examined how they themselves felt about it. In the 
process of conducting it, they also found research "is a lived practice" that is shaped by 
and shapes the researcher (Sumara & Carson, 2001 , p. xiii). 

When the students received their contact lists for carrying out the evaluation project, they 
took pause to think of how to best to make the process rewarding for both actors in the 
process, given what they now knew about the project. The eight student evaluators first 
interviewed each other in pairs, using the questionnaire, to get in gear and note areas that 
might offer problems or needed clarification in obtaining qualitatively measurable results. 
This process led them to think more deeply about their own responses while explaining 
them face-to-face. Writing a reaction is one thing, and developing a larger picture orally 
is another. A little gentle prodding often summoned up a story that served to strengthen 
an observation or sharpen a point. 

Then it was time to contact the people on their list. Typically an interview was initiated 
by thanking the participant for their time, thoughts and candor to start the session on a 
appreciative note. Next, the main themes of the assessment were reiterated to set the 
stage for the individual questions: we were looking for positive perceptions and 
experiences in this educational program and recommendations for what they wanted 
more of now and in the future. After all, as Program enrollees and graduates, they were 
the best source to quarry for answers about how well the Program was serving its target 
audience and larger society in general. Were their objectives being met by what they 
learned or were learning at present? If they had concerns about aspects of the program, 
could they pose some constructive wishes to improve upon it, wave a magic wand so to 
speak? In this respect, their reponses could emphasize positive ways to strengthen the 
Program rather than negative views of problems to be solved. 
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Although apprehensive at first (after all, why would someone want to share their 
experiences with someone they had never met?), the students conducting the evaluation 
were impressed and delighted by the stories they heard, especially those about peak 
experiences in the Program and how learning was already enhancing daily lives. For 
example, one current student was directly applying the methodology she had learned to a 
corporate training program in a national travel firm. Others were using the concepts and 
techniques in their leadership efforts in introducing and enhancing computer-based 
learning in K-12 classrooms. The student evaluators’ mental models were being changed 
and elaborated with each story, and the Program became more of a living system in 
relation to real-life projects and processes. We connected with each other and with our 
course work more and more as the interviews proceeded, and the appreciative nature of 
our investigation brought forth positive feelings and perceptions that otherwise may not 
have been so strongly sensed and expressed. 

As we probed our fellow students' feelings about objectives and future goals and how the 
Program was meeting their needs and expectations, we formed a clearer vision of what 
the Program could and should be. We found out that "educational technology" meant 
many things to many people. There was an expressed need for becoming conversant with 
technical trends in education and how best to integrate new tools and methods into the 
classroom or other learning environments. Many saw educational technology as a wave 
of the future, and wanted to increase their comprehension of a new phenomenon in 
general and their practice of it in particular. Many were continuing their education at a 
distance from the university, and were very grateful to have the courses offered at a 
distance, with occasional face-to-face meetings with classmates and faculty at a 
designated learning center near them during the semester. 

Several themes emerged. The participants valued the availability and flexibility of 
courses, and the transferability and marketability of skills they were mastering. Those at 
a distance and with employment or family obligations especially appreciated instructors 
who devoted extra effort to accommodate their needs. The passion of instructors who 
dedicated themselves to their knowledge field and student needs was cited as a core 
characteristic. Many felt that they were keeping on the edge of technology with the 
Program and were excited about the group interactions they were having in both face-to- 
face and virtual classrooms. The participants' stories about dealing with their own 
students, families and career obligations brought the academics of the Program alive: we 
all felt part of a mission that was being successfully played out as we completed personal 
life goals. Hence its narrative feature could have made the assessment itself a highlight 
of the Program for the student evaluators. 

It was gratifying to hear eager replies denoting an authentic willingness to participate in a 
study which could serve to enhance the efforts that all stakeholders were putting forth in 
the interests of educational excellence in the Department: administration, staff, faculty 
and students alike. Perhaps this was because the conversation was student-to-student 
rather than student-to-faculty, allowing for unfettered dialogue. Interviewees candidly 
voiced their concerns in constructive ways to suggest improvement in many areas, 
trusting their views would be respected and taken seriously by the Division. Because the 
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students conducting the interviews were also interviewees themselves, they could be 
regarded as facilitators of the process rather than authority figures. We were all in this 
together, and fellow students were trusted to represent the findings of the evaluation 
accurately and fairly. The action research nature of the study gave everyone expectations 
that the outcomes would have a positive impact on the methods and content of the 
Program. 

The Faculty Response 

While all eight student evaluators conducted interviews, analyzed the data, and wrote the 
evaluation report, a smaller group presented the recommendations from the project to the 
Program faculty. The faculty were extremely attentive and inquisitive about the details of 
the findings as the student presenters alternatively explained sections of the report and 
answered questions. The action research had unearthed student views they had not before 
heard with regard to some aspects of the program: advantages and limitations of distance 
classes, significance of hands-on experience in various technologies, and so on. The very 
meaning of "educational technology" in general could widely vary among students who 
had entered the Program with differing knowledge levels and objectives. 

The Appreciative Inquiry approach revealed the Division was doing many things right 
and needed to continue to build on those strengths. It also revealed areas that were in 
need of improvement. Several recommendations were made to the faculty. A meeting 
was held approximately one month after receiving the Report Recommendations where 
the faculty identified actions to take and commitments to make. 

The first action taken was to revisit and revise the mission statement of the Division to 
reflect the continuing philosophy of teaching concepts over specific skills. The revised 
mission statement was shown to students in a summer course, all of whom had conducted 
the Program evaluation or participated in it. They agreed the new statement did a better 
job of conveying the true intent of the Program. Several other actions and commitments 
were made. It was agreed that an Orientation Program would be scheduled during the 
first week of each semester. Starting a web-based newsletter was considered. The 
newsletter could be a promotion and communication tool as well as provide a place to 
highlight accomplishments of students (current and former) as well as faculty. Instituting 
brown bag lunch seminars (primarily for faculty but open to students) was also discussed. 
The brown bag lunch seminars would be used for faculty members to share their 
expertise with one another as well as further each other's learning by keeping current in 
the field. It was agreed that a face-to-face component would be part of every class as 
much as possible; however, it was also acknowledged that due to limited resources, it 
might not be possible to offer each course in both distance and face-to-face delivery 
formats. 




10 



10 



What We Learned 



On the surface, it would appear that by framing questions positively (which is a hallmark 
of AI), problems and concerns would be glossed over. However, this was not the case. 
We found that when students had a concern, they voiced it. The difference in using the 
AI approach is that instead of dwelling on the “problem,” the conversation focuses on 
suggestions for what could be done about it. A generative energy is created as 
possibilities unfold. We all emerged with a far better view of what the Program was 
accomplishing, where it might be falling short, and what could make it even better. In 
informing each other through positive questions and responses, we became excited and 
energized to put forth our best efforts on all fronts toward achieving excellence. In this 
process, it became clear that the Program could be strengthened by keeping what was 
valued, discarding what was not valued, and creating what did not currently exist but was 
envisioned by both faculty and students. 

Ludema (2000) tells us that “social knowledge and organizational destiny are tightly 
intertwined. . .all inquiry into organizational life should be collaborative” (p. 281). This 
project as a whole can be regarded as a collaborate effort at constructing knowledge of 
what the Division was attempting to achieve and how well program goals were being met 
by all involved. The student evaluators and students interviewed engaged in a dialogue 
from which new understanding was reached about their own and others’ roles in making 
the Program a success. Employing the "art and science of asking powerful, positive, 
questions"(Cooperrider, 2000, p. 123) led us away from a path of negativity, criticism 
and "spiraling diagnosis" and to a path of "discovery, dream, and design" (Cooperrider, 
2000, p. 124). 

The combination of AI, narrative inquiry, and action research elicited stories rich in data. 
Those who participated in the Program evaluation willingly shared their stories. 
Embedded in the stories were practical suggestions and recommendations for the 
stakeholders (faculty and students) of the Program. While serious issues surfaced and 
were addressed, valuable information, key insights, and pleasant surprises made this an 
evaluation report that was acted upon rather than relegated to a bookshelf to collect dust. 
Perhaps this is evidence that supports Davenport and Prusak's statement: "Human beings 
learn best from stories" (1998, p. 81). 



Appreciative Program Evaluation in Higher Education 

What we choose to study and measure in the evaluation process is a sign of what is 
valued in the system. We typically use the evaluation process to identify what we need 
less rather than more of. The AI approach inquires into a program's successes and in 
practice, is an inquiry into discovering what gives life to that system. A basic tenant of 
AI is that a system will move in the direction of what it is studying. If this is true, what 
we choose to evaluate and how we evaluate it becomes fateful: 
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When we inquire into the things in our organisations that are life giving, we begin 
to understand that we can choose to focus on those qualities. Through asking 
others to join in our inquiry, we can have a considerable impact on the image of 
our organisation and, ultimately, on the way it functions (Mohr, Smith & Watkins, 
2001, p. 315). 

Program evaluation is something that we are all called upon to do in Higher Education. 
Often, the evaluation focuses on problems and what is wrong with the program. 
Appreciative Inquiry offers an approach that highlights what the program is doing right 
and what it could be doing “more” of. It may provide a particularly useful framework for 
monitoring courses and has, in fact, been used by one of the authors for the traditional 
end-of-the-semester-course evaluation (Norum, 2001). Concerns and problems will still 
be identified. The refreshing aspect is that instead of dwelling on those “problems,” a 
creative, generative energy is engendered. By identifying what is being done “right,” 
programs can be strengthened by keeping what is currently valued, discarding what is not 
valued, and creating what does not currently exist but is envisioned by both faculty and 
students. 
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